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(1)  Summary  of  Technical  Results 

The  support  provided  under  Office  of  Naval  Research  Contract  Number  N00014- 
88-K-0632,  Programming  with  Articulated  Objects,  has  enabled  us  conduct  important 
research  in  programming  methods  for  the  design,  analysis,  and  control  of  mechanical 
devices.  The  contract  also  supported,  in  part,  the  development  of  new  techniques  for 
image-based  motion  analysis.  Work  supported  by  the  project  was  organized  under  four 
projects.  A  brief  summary  of  the  accomplishments  for  each  project  is  described  below. 
The  attached  publications  and  reports  provide  more  detailed  presentations  of  the 
results.  Additional  publications  are  currently  in  preparation  and  will  be  forwarded 
upon  completion. 
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Project  1:  Programming  Mechanical  Simulations 

'z  Computer  simulation  promises  to  have  a  dramatic  influence  on  the  study  of 
mechanical  systems  and  the  interaction  between  humans  and  machines.  The  central 
goal  of  this  project  was  to  study  methodologies  for  controlling  complex  physical 
objects  in  simulation.  Our  work  provides  a  conceptual  framework  that  integrates  the 
mechanical  and  control  components  of  a  simulation.  The  programming  paradigm  that 
forms  the  basis  of  this  framework  views  a  mechanical  simulator  as  a  multi-level  con¬ 
straint  solver.  The  usefulness  of  the  approach  was  demonstrated  through  a  series  of 
experiments  examining  the  control  of  robot  locomotion.  t  . , 

Constraint-oriented  Simulation  — 

In  our  work,  simulation  is  cast  as  a  problem  of  solving  a  sequence  of  time-varying 
dynamic  constraints  on  the  attributes  of  physical  objects.  Constraints  due  to  mechani¬ 
cal  considerations,  such  as  joints  between  bodies,  and  constraints  due  to  control  con¬ 
siderations,  such  as  guided  movementsT are  treated  in  a  uniform  fashion.  At  any  instant 
in  time,  a  system  of  linear  equations  constrains  the  the  instantaneous  motions  of  bodies. 
These  motion  equations^relate  accelerations  of  bodies  to  the  forces  and  torques  on  the 
bodies.  Given  the  positions  and  velocities  of  all  bodies  at  time  t,  positions  and  veloci¬ 
ties  at  time  t  +  Arare  found  by  solving  the  motion  equations  for  accelerations  and 
integrating  from  previous  states.  Given  initial  positions  and  velocities,  this  process  is 
iterated  to  simulate  motion  over  an  interval  of  time. 


The  set  of  motion  constraint  equations  is  derived  from  three  sources.  First,  a  base 
set  of  equations  modeling  the  fundamental  laws  of  motion  and  the  mechanical  connec¬ 
tions  between  objects  is  automatically  formulated  from  a  model  definition.  Secondly, 
the  base  set  of  equations  may  be  reformulated  during  the  course  of  a  simulation  to 
accommodate  changing  relationships  among  objects  as  new  contacts  are  made  and  old 
contacts  are  broken.  Lastly,  control  routines  may  influence  the  composition  of  the  set 
of  motion  equations.  The  user  conducting  the  simulation  models  control  constraints  by 
programmatically  adding  and  deleting  named  equations  to  the  set  of  motion  equations. 


In  many  simulation  systems  including  ADAMS,  DADS,  and  SD/FAST,2’7, 14, 16 
motion  constraints  are  predetermined  by  a  model  of  the  mechanism  and  remain 
unchanged  during  simulation.  By  allowing  conditional  reformulation  of  motion  con¬ 
straints  and  by  incorporating  control  constraints  into  the  set  of  motion  equations,  the 
power  and  expressiveness  of  simulation  programs  is  significantly  extended.  However, 
the  constraint  restructuring  must  be  accomplished  in  a  carefully  disciplined  manner  to 
avoid  comprising  the  integrity  of  the  simulation.  We  have  developed  a  regimen  under 
which  the  flexibility  of  constraint  editing  is  achieved  without  disrupting  the  basic 
integration  process. 


Object-level  Programming  for  Locomotion 


We  demonstrated  our  control  methods  by  simulating  robot  locomotion.  Using  the 
constraint- based  programming  paradigm,  we  derived  model-independent  control  algo¬ 
rithms  for  hopping.  We  have  successfully  simulated  hopping  for  a  variety  of  designs. 
Our  approach  concentrates  on  the  interaction  between  the  object  and  its  surroundings 
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at  points  of  contact.  It  is  only  through  reactions  at  contact  points  that  a  hopper  can 
alter  it  momenta.  Control  programs  are  expressed  as  dynamic  constraints  on  the  exter¬ 
nal  forces  and  torques  applied  to  the  hopper  by  pushing  against  surfaces  in  contact. 

The  nature  of  contact  relations  places  strict  constraints  on  the  dynamics  of  control 
problems.  Control  is  organized  as  a  two-step  process.  Within  the  bounds  of  contact 
constraints,  a  plan  for  applying  forces  and  torques  to  the  whole  hopper  is  devised.  The 
use  of  a  generalized,  virtual  leg  permits  programs  to  express  the  motion  of  hopping 
bodies  independent  of  their  structure.  Secondly,  the  controllable  degrees  of  freedom 
are  used  to  achieve  the  required  contact  forces. 

The  resulting  programs  are  comparable  to  object-level  programs  for  robot  manipu¬ 
lation.  An  object-level  language  specifies  robot  operations  by  defining  the  desired 
state  of  the  object  to  be  manipulated.3, 22,23,29,30, 32  The  robot  actions  required  to 
bring  about  the  necessary  changes  to  the  object  are  determined  by  lower  levels  of  the 
robot  system.  In  a  similar  way,  contact  constraints  treat  the  robot  as  an  object  to  be 
pushed  or  spun.  The  actuator  values  required  to  achieve  a  desired  external  force  or 
torque  are  determined  by  lower  level  processes. 

To  demonstrate  the  generality  of  our  approach,  we’ve  tested  a  variety  of  hoppers 
using  the  same  high-level  control  program.  The  two  basic  designs  we’ve  used  are 
shown  in  Figure  1.  The  hopper  on  the  left  models  the  CMU  one-legged  hopper.31  The 
hopper  on  the  right  emulates  the  torso,  upper  leg,  and  lower  leg  of  an  anthropoid.  A 
single  hop  by  the  anthropomorphic  hopper  is  shown  in  Figure  2.  Both  hoppers  were 
robustly  controlled  by  the  same  set  of  contact  constraints. 

The  usefulness  of  contact  constraint  programming  goes  beyond  basic  hopping.  A 
simple,  one-legged  robot,  such  as  the  CMU  hopper,  has  contact  with  the  ground  during 
stance  at  a  single  point.  The  point  contact  permits  only  a  single  constraint  force  to  be 
applied  to  the  hopper.  For  this  reason,  the  external  force  and  moment  on  the  hopper  are 
intrinsically  coupled.  More  complex  interactions  afford  a  rich  set  of  alternatives  for 
control  required  to  achieve  many  behaviors.  For  example,  a  hopper  with  a  sizable  foot 
touching  the  ground  shares  a  plane  of  contact  with  the  support  surface.  With  planar 
contact,  the  hopper  can  independently  control  the  external  force  and  moment  during 
stance.  We’ve  demonstrated  the  increased  controllability  by  simulating  a  flip  with  no 
change  in  horizontal  velocity.  Because  the  external  force  and  moment  are  decoupled, 
the  hopper  can  jump  vertically  with  sufficient  momentum  to  rotate  through  a  complete 
circle.  We  believe  that  the  contact  constraint  analysis  will  prove  useful  in  understand¬ 
ing  a  wide  variety  of  self-movement  tasks. 

Technical  Reports 

J.K.  Kearney  and  D.H.  Lee,  "Motion  control  with  process  abstraction,''  Third  IEEE 
Intel national  Symposium  on  Intelligent  Control,  Aug.  1988. 

J.K.  Kearney  and  S.  Hansen,  Generalising  the  Hop:  Object-level  programming  for 
legged  motion"  submitted  to  The  International  Journal  of  Robot  ics  Research,  also  pub¬ 
lished  as  University  of  Iowa  Tech  Report  UI-CS  90-09 
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Project  2:  Stream  Editing  for  Animation 

The  first  step  in  creating  a  computer  animation  is  often  the  definition  of  a  time- 
varying  geometric  model.  Powerful  motion  generation  tools  enable  designers  to  create 
physically  realistic  and  biologically  plausible  object  motions  through  dynamic  simula¬ 
tion  and  constraint-based  optimization.4'6,  15,24,25,33  Motion  models  can  also  be 
determined  from  measurements  of  physical  motions  using  techniques  of  cinematogra¬ 
phy,  photogrammetry,  and  accelerometry.  This  project  investigated  methods  for  com¬ 
bining  and  modifying  motion  sequences  and  for  integrating  viewing  models  with  object 
movements. 

Our  approach  is  based  on  the  representation  of  movement  sequences  as  streams. 
We  have  developed  software  that  allows  streams  to  be  filtered,  duplicated,  transformed, 
and  combined.  Cameras  with  prescribed  motions  can  be  introduced  to  visualize 
scenes.  A  general  facility  for  composing  sequences  of  spatial  transforms  permits  the 
specification  of  compl  * .  trajectories  and  relative  motion. 

The  resulting  motion  sequences  can  be  rendered  as  animation  or  presented  as  the 
input  circumstances  for  simulation.  Man-in-the-loop  simulation  offers  enormous 
potential  for  testing  and  training  of  human  operators  in  hazardous  situations  such  as 
flying,  driving,  and  teleoperation  of  robots.  The  synthesis  of  realistic  situations  is  criti¬ 
cal  for  the  effective  development  of  computer  simulated  training  and  testing  applica¬ 
tions.  Streams  provide  a  conceptual  framework  for  composing  complex  dynamic  set¬ 
tings  and  a  computational  model  that  is  robust,  reliable,  and  simple  to  implement. 

We  present  a  simple  example  of  stream  manipulation  to  illustrate  how  a  single 
stream  acquired  from  a  physically-based  simulation  of  walking  can  be  duplicated, 
transformed,  and  assembled  into  a  new  stream  that  represents  a  precision  drill  team 
marching  in  formation.  Our  starting  point  is  a  stream  that  models  a  walking  figure. 
The  stream  was  produced  by  the  physical  simulator  newton.  To  produce  a  second 
walking  figure  whose  path  is  displaced  from  the  first,  we  create  a  transformed  copy  of 
the  walking  stream.  The  two  streams  are  merged  to  create  a  stream  with  a  pair  of 
figures,  walking  in  step  with  one  another.  The  pair  of  walking  figures  can  be 
transformed  and  merged  to  obtain  a  quartet  of  walkers.  This  process  can  be  repeated  a 
number  of  times  to  produce  a  whole  brigade.  The  brigade  is  represented  as  a  single 
stream  that  can  be  reproduced,  reoriented,  repositioned,  and  combined  with  the  original 
marchers  to  create  two  divisions  marching  in  different  directions.  With  a  judicious 
selection  of  the  transformation  parameters,  we  can  have  the  two  formations  cross  paths, 
with  members  of  each  brigade  passing  between  members  of  the  other. 

A  scene  from  the  motion  sequence  with  two  groups  of  four  walkers  is  pictured  in 
Figure  3.  The  operations  required  to  assemble  the  this  model  are  schematically  shown 
in  Figure  4.  Edges  indicate  data  streams  and  boxes  indicate  stream  producing  opera¬ 
tions.  One  of  our  long  range  goals  is  to  build  a  graphic  programming  language  similar 
to  these  schematic  diagrams. 

The  marching  example  illustrates  the  simplicity  and  power  of  animation  stream 
editing.  The  niarchine  example  ran  be  constructed  even  more  compactly  than 
presented  above  oy  using  transform  generators  that  allow  sequences  of  transforms  to  be 
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defined.  Stream  operations  focus  attention  on  the  flow  of  data  through  computational 
processes.  Successive  members  of  the  stream  funnel  through  filters  or  functions  that 
are  mapped  over  the  stream.  As  a  consequence,  graph  representations  such  as  the  one 
shown  in  Figure  4  are  well  suited  for  visualizing  networks  of  stream  operations. 

The  focus  of  our  work  has  been  on  the  motion  of  bodies  with  rigid  members  and 
cameras.  However,  the  approach  could  easily  be  adapted  to  include  other  types  of 
objects  that  undergo  parametric  variation  over  time.  Moving  light  sources  with  chang¬ 
ing  brightness  or  chroma  could  be  added.  Deformable  objects  which  bend  or  grow 
could  also  be  included. 

Technical  Reports 

J.k.  Kearney  and  S.  Hansen,  "Stream  Editing  for  Animation"  submitted  to  The  Visual 
Computer,  also  published  as  University  of  Iowa  Tech  Report  UI-CS  90-08. 
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Project  3:  Optimal  Motion  Analysis 

This  project  investigated  integrated  environments  for  optimal  motion  analysis. 
Optimal  motion  analysis  is  of  interest  for  the  development  of  efficient  control  algo¬ 
rithms  for  robots,17  to  maximize  athletic  performance,  and  to  improve  rehabilitoTy 
treatment.21  Optimal  motion  generation  has  also  been  proposed  as  a  means  to  create 
realistic  animations  of  complex  movements.33  A  large  number  of  studies  have 
observed  stereotypic  motion  patterns  among  humans.21  It  has  long  been  conjectured 
that  the  motions  naturally  preferred  by  humans  for  walking,  throwing,  jumping,  and 
running  are  minimal  with  respect  to  some  objective  function.  Biologists  still  debate 
over  what  the  underlying  objective  function  or  functions  might  be.  Optimal  motion 
analysis  provides  a  constructive  means  to  test  these  hypotheses. 

A  model-driven  system  for  motion  optimization  named  Optimizer  was  completed 
in  the  last  year  of  the  contract.  Optimizer  allows  a  user  to  simply  define  planar  chains 
of  rigid  links.  The  desired  motion  of  an  object  is  specified  by  imposing  constraints  on 
the  state  of  object.  At  the  user’s  request,  the  system  will  determine  a  trajectory  that 
satisfies  these  constraints  and  is  optimal  with  respect  to  some  objective  criteria.  The 
time-varying  values  of  important  state  variables  can  be  conveniently  inspected  and 
results  can  be  recorded  for  later  study  and  comparison. 

Optimizer  presents  a  uniform  view  of  the  system  to  the  user.  A  common  language 
interface  is  used  to  define  models,  to  specify  constraints,  and  to  inspect  experimental 
results.  In  addition  to  the  language  interface,  menus  provide  a  convenient  means  to 
navigate  through  the  various  design  functions.  The  menu  interface  provides  a  user 
friendly  environment,  and  the  user  can  exploit  the  power  of  expressions  provided  by 
the  language  interface  to  work  with  the  system. 

Optimizer  solves  for  optimal  motions  by  discretizing  a  continuous  motion  into  a 
sequence  of  time-varying  positions.  The  granularity  of  the  discretization  can  be 
selected  by  the  user.  Given  an  initial  sequence.  Optimizer  formulates  a  nonlinear  least 
squares  problem  which  is  solved  with  a  version  of  the  Levenberg-Marquardt  algorithm. 

The  principal  body  of  Optimizer  was  programmed  in  Lisp.  Development  was 
facilitated  by  incorporating  a  diverse  set  of  software  packages  to  perform  many  of  the 
required  support  functions.  The  special  purpose  application  languages  for  object 
definition,  constraint  specification,  and  report  generation  were  parsed  using  the  UNIX 
utilities  lex  and  yacc.  User  programs  were  translated  into  Lisp  functions  and  executed. 
Symbolic  expressions  for  the  equations  of  motions  were  derived  from  the  object  model 
using  the  symbolic  mathematics  package  Macsyma.1  The  motion  equations  derived  in 
Macsyma  and  motion  constraints  specified  by  the  user  were  combined  and  translated 
into  a  Fortran  function,  fcn(),  that  encapsulated  the  objective  function.  Given  values 
for  all  joint  positions  not  specified  by  the  problem  constraints,  this  function  returned 
the  torque  at  each  joint  at  each  time  step.  The  set  of  unconstrained  joint  positions  that 
minimized  sum  of  squared  joint  torques  was  determined  with  a  robust  Fortran  routine 
lmdiflf)  from  the  MINPACK  subroutine  package.11  Graphics  routines  were  borrowed 
from  the  simulator  Newton.11  Optimizer  demonstrates  the  range  of  support  functions 
required  for  integrated  design  environments  and  the  importance  of  incorporating 
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softwared  packages  for  complex  engineering  applications. 

To  illustrate  the  use  of  Optimizer,  we  present  the  results  of  an  analysis  of  the 
swinging  motion  of  a  golfer.  The  model  consists  of  two  links.  One  link  represents  the 
golfer’s  arms  and  the  second  link  represents  the  golf  club.  The  link  representing  the 
arms  is  constrained  to  rotate  about  a  fixed  axis  located  at  the  shoulder.  The  motion  is 
initiated  with  the  joints  positioned  to  begin  the  forward  swing  of  the  club  and  with  no 
angular  velocity.  The  final  configuration  constrains  the  club  head  to  be  in  contact  with 
a  ball.  The  final  joint  velocities  are  specified  such  that  the  club  head  has  a  horizontal 
velocity  while  striking  the  ball.  Figure  5  illustrates  the  solution  of  the  golf  swing 
optimization.  Link  positions  are  overlayed  in  the  window  located  upper-left  side  of  the 
screen.  Time-varying  values  of  state  variables  are  displayed  in  the  graphs  on  the  right 
side  of  the  screen.  The  motion  sequence  qualitatively  corresponds  to  motions  reported 
in  experiments  with  real  golfers.10, 19 

Technical  Reports 

B.  Prasad,  "Optimizer:  A  Model  Driven  System  for  the  Study  of  Optimal  Motions," 
MA  thesis.  University  of  Iowa,  August,  1990. 
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Project  4:  Image  Correspondence  for  Particles 

Accurate  measurement  of  particle  location  and  motion  is  important  for  a  diverse 
assortment  of  scientific  and  engineering  problems  including  the  design  of  ship  hulls, 
automobile  cylinders,  and  heart  valve  replacements.  A  particle  is  a  small,  compact 
object  such  as  an  oil  droplet,  a  blood  cell,  or  a  piece  of  dust.  Fluid  motions  can  be 
quantitatively  assessed  by  tracking  particles  seeded  m  flow  fields.  The  three- 
dimensional  distribution  of  particles  at  an  instant  in  time  can  also  give  important  infor¬ 
mation  about  the  characteristics  of  gas  and  fluid  motions.  This  project  examined  the 
problem  of  measuring  the  location  of  large  numbers  of  particles  at  a  snapshot  in  time 
using  image  processing  techniques.  The  algorithm  we  devised  can  be  simply  extended 
to  measure  particle  motions. 

Information  about  the  three-dimensional  location  of  an  object  can  be  recovered 
from  multiple  images  of  the  object  taken  simultaneously  from  different  viewpoints.9 
This  process  is  usually  performed  in  three  steps.  First,  sets  of  object  features  are 
independently  found  in  each  image.  An  object  feature  is  a  visually  distinctive  charac¬ 
teristic  that  can  be  reliably  detected  from  different  viewpoints.  The  edges  and  comers 
of  surface  markings  are  often  used  as  features.  Next,  features  are  matched  across  the 
images.  The  last  step  is  to  estimate  the  three-dimensional  location  of  the  feature  by  tri¬ 
angulation.  The  use  of  two  or  more  views  to  infer  three-dimensional  structure  and 
shape  is  called  stereo  vision.  Considerable  research  has  been  directed  towards  methods 
that  rely  on  two  cameras.  Recently,  a  number  of  papers  have  pointed  out  the  advan¬ 
tages  of  introducing  a  third  camera.1 2’ 13, 18,20,26-28,34-36 

This  project  addressed  the  matching  problem  for  particles.  The  particles  to  be 
viewed  were  assumed  to  be  small  spheres  with  a  size  near  the  resolution  limit  of  the 
camera.  For  our  model,  we  assumed  that  particles  appeared  as  points  in  an  image.  Our 
problem  was  to  find  points  in  two  or  more  different  images  that  were  projections  of  the 
same  particle.  This  presents  a  difficult  matching  problem,  because  all  image  points 
look  alike.  Further,  neighboring  image  points  may  lie  at  disparate  depths.  As  a  conse¬ 
quence,  stereo  matching  algorithms  that  rely  on  surface  continuity  or  the  appearance  of 
distinctive  markings  are  inappropriate  for  pm  tide  matching. 

We  devised  a  general  algorithm  for  matching  image  points  in  three  views.  The 
method  relies  solely  on  geometric  constraints  based  on  the  camera  arrangement.  In 
previous  formulations  of  the  trinocular  matching  algorithm,  one  image  was  used  to  ver¬ 
ify  the  consistency  of  possible  matches  between  points  in  the  other  two  images.  Our 
approach  generalizes  this  strategy  by  considering  all  pairwise  matches  between  points 
in  all  images.  We  formulate  the  matching  problem  as  a  search  for  edges  contained  in 
all  perfect  matchings  of  a  bipartite  graph.  The  bipartite  graph  links  points  between  two 
images  that  satisfy  a  geometric  consistency  test.  To  take  maximal  advantage  of  the 
geometric  constraints,  three  such  problems  must  be  solved.  Our  algorithm  employs  an 
iterative  search  to  find  certain  matches.  The  algorithm  has  been  tested  with  syntheti¬ 
cally  generated  particle  images.  The  results  show  that  it  finds  significantly  greater 
numbers  of  matches  than  earlier  methods  when  matching  uncertainty  is  high. 
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An  attractive  property  of  our  algorithm  is  its  graceful  degradation  in  response  to 
image  distortion  and  modeling  error.  Although  measurement  errors  may  prevent  suc¬ 
cessful  matching,  wrong  matches  can  almost  always  be  avoided  if  the  error  in  the 
image  position  of  a  particle  can  be  bounded.  Thus,  noise  can  cause  a  loss  of  acuity  but 
should  not  cause  the  introduction  of  gross  misinformation  that  could  result  from 
incorrect  matching.  The  density  of  matches  obtained  by  the  method  can  be  optimized 
by  regulating  the  number  of  particles  seeded  in  the  volume  and  by  selectively  arrang¬ 
ing  the  cameras  to  minimize  ambiguity  in  the  matching  process.  The  method  can  be 
simply  extended  to  match  image  streaks  formed  by  using  long  exposure  views  of  mov¬ 
ing  particles. 

Technical  Reports 

J.K.  Kearney,  X.  Yang,  and  S.  Zhang,  "Camera  calibration  using  geometric  con¬ 
straints,"  IEEE  Conference  on  Computer  Vision  and  Pattern  Recognition,  June  i 989, 
pp.  672-679. 

J.K.  Kearney  "Trinocular  Correspondence  for  Particles  and  Streaks,"  submitted  to  the 
IEEE  Conference  on  Computer  Vision  a>id  Pattern  Recognition. 
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Figure  1.  Two  hopper  designs.  The  model  on  the  left  is  an  idealization  of  the  CMU 
one-legged  hopper.  The  model  on  the  right  empulates  an  anthropoid. 


Figure  2.  An  overlayed  sequence  of  frames  from  a  single  hop  with  the  anthropo¬ 
morphic  hopper.  The  longitudinal  axis  of  each  link  is  drawn. 


Figure  3.  Selected  scenes  from  a  motion  sequence  with  eight  walking  figures  moving 
in  groups  of  four.  The  scenes  are  ordered  clockwise  beginning  with  the 
upper,  left  panel.  The  sequence  was  created  using  the  stream  editing  pro¬ 
gram  diagrammed  in  Figure  4. 


walking-motion-stream 


Figure  4.  A  schematic  representation  of  the  stream  editting  program  to 
create  8  marchers. 


Figure  5.  The  optimal  motion  for  a  two-link,  planar  manipulator.  The  linkage  was 
constrained  to  have  zero  initial  velocity.  Final  joint  velocities  were  con¬ 
strained  so  that  tip  moved  horizontally.  Graphs  of  motion  parameters 
appear  on  the  right.  The  top  four  panels  shown  plot  of  joint  angles  over 
time.  Initial  values  are  shown  on  the  right;  final  values  are  shown  on  the 
left.  The  bottom  two  panels  give  the  torque  graphs  for  the  estimated 
optimal  motion. 
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